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* NOTICES * 




JPO and NCI PI are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2 **** s hows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The procedure of receiving the character string of the natural language expressing a demand 
from a predetermined input means, While choosing the optimal software object for performing the 
procedure which analyzes the semantics of the language which said character string shows, or a 
sentence, and generates semantic representation, and processing according to said demand based on said 
semantic representation The procedure of setting up the environment for operating this software object, 
and said semantic representation are translated. The procedure which generates the functional 
description expression which consists of the language which normalized the operating instructions 
which should be given to this software object in order to make said software object perform processing 
according to said demand, The procedure which generates the instruction which can perform said 
software object from said functional description expression, and is transmitted to this software object, 
And operating instructions of the software object by natural language characterized by making a 
computer perform processing including the procedure which outputs the result of the processing which 
said software object performed in response to said instruction with the predetermined gestalt which said 
user can recognize. 

[Claim 2] The program for making a computer perform processing according to an approach according 
to claim 1 . 



[Translation done.] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the method of operating the software object which 
operates on a computer with natural language, and the program for it. In this specification, a software 
object means the operating system (OS) for controlling electronic equipment, such as a personal 
computer and a microcomputer control equipment, and the thing of an application program which 
operates on OS. Moreover, the signal received from input devices (a keyboard, a microphone, 
handwriting tablet, etc.) is processed, the character string of natural language is generated, the character 
string is analyzed, and the thing of a system which generates the operating instructions of a software 
object based on an analysis result is called a natural language interface on these specifications. 
[0002] 

[Description of the Prior Art] Research on the natural language interface for operating a software object 
with natural language is done more briskly than before. As an example, the handwriting input approach 
given in JP,8- 147096, A and equipment, an information processor given in JP,6-75692,A, an information 
input unit given in JP,6-131 108, A, an information input unit given in JP,6-282566,A, etc. are 
mentioned. The natural language interface of these former is used in order to call the internal function 
prepared for the software object with natural language. For example, a word processor with which the 
character string beforehand specified that a user does a handwriting input to "It enlarges" becomes a 
double width character is indicated by JP,6-75692,A. Moreover, if a user does a handwriting input with 
an "image transcription", the videocassette recorder which has a control system from which image 
transcription processing begins is indicated by JP,8- 147096, A. 
[0003] 

[Problem(s) to be Solved by the Invention] The conventional natural language interface is only what 
specialized in each software objects, such as for example, a word processor program and a control 
program of a videocassette recorder, and diverting the natural language interface developed for a certain 
software object to actuation of other software objects is not assumed fundamentally. For this reason, a 
software-development person has to spend a considerable effort to equip the natural language interface 
for some software objects, in order to newly develop the natural language interface of dedication. 
[0004] Moreover, in the conventional natural language interface, it is assumed that the instruction for 
calling the internal function beforehand prepared for the software object is inputted. Therefore, the 
software object is equipped with what kind of function, and the user has to have beforehand the 
information (knowledge) about with what kind of natural language it should be called with him. That is, 
to compensate for the function of a software object, a user issues an instruction rather than a software 
object functions in the form which meets the demand of a user. As long as it stops at such a gestalt, 
actuation of the software object by natural language cannot but become a thing lacking in flexibility. For 
example, suppose that the user thought "I want to create information of a movie appreciation meeting", 
and inputted by making the idea into language as it is. It is not the instruction for "I want to create 
information of a movie appreciation meeting", expressing a demand of a user, a wish, or the purpose so 
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to speak, and calling a certain^fction of a software object clearly. It is i^^sible to perform suitable 
processing to such an input in the conventional natural language interface. 

[0005] The place which accomplishes this invention in order to solve the above technical problems, and 
is made into the purpose is to offer the technique for realizing the natural language interface equipped 
with the flexibility which can perform suitable processing, even when the natural language expression of 
a demand of a user, a wish, the purpose, etc. is thought as an input to be the versatility which can operate 
systematically the software object from which plurality differs. 
[0006] 

[Means for Solving the Problem] The operating instructions of the software object by the natural 
language concerning this invention accomplished in order to solve the above-mentioned technical 
problem The procedure of receiving the character string of the natural language expressing a demand 
from a predetermined input means, While choosing the optimal software object for performing the 
procedure which analyzes the semantics of the language which said character string shows, or a 
sentence, and generates semantic representation, and processing according to said demand based on said 
semantic representation The procedure of setting up the environment for operating this software object, 
and said semantic representation are translated. The procedure which generates the functional 
description expression which consists of the language which normalized the operating instructions 
which should be given to this software object in order to make said software object perform processing 
according to said demand, The procedure which generates the instruction which can perform said 
software object from said functional description expression, and is transmitted to this software object, 
And it is characterized by making a computer perform processing including the procedure which outputs 
the result of the processing which said software object performed in response to said instruction with the 
predetermined gestalt which said user can recognize. 

[0007] Moreover, this invention offers the program for making a computer perform the above 

processings. 

[0008] 

[Embodiment of the Invention] It explains concretely, referring to a drawing about the procedure of 
processing by this invention. 

[0009] First, the character string which a user inputs with natural language using the input means of a 
computer is received (step 50). Hardware, such as a keyboard, a handwriting input unit, and an audio 
input unit, and the software (a keyboard driver, pattern recognition software, speech recognition 
software, etc.) which generates the character string of natural language from the output signal of the 
hardware constitute an input means. Here, the character string of "wanting to create information of a 
movie appreciation meeting" should be inputted. 

[0010] Next, the character string generated as mentioned above is analyzed, and semantic representation 
is generated (step 51). This processing can be performed by the natural-language-processing approach of 
common knowledge including procedures, such as morphological analysis, syntax analysis, and a 
semantic analysis, here, the semantic representation "a movie (that)", "an appreciation meeting (that)", 
"information (**)", and "creation (I want to carry out)" should be generated 

[001 1] Next, the optimal software object for performing processing according to a demand of a user is 
chosen based on the above-mentioned semantic representation (step 52). Selection of a software object 
is performed using the dictionary (it is hereafter called a configuration section dictionary) which 
associated semantic representation and a software object. An example of a configuration section 
dictionary is shown in drawing 2 . evaluating point =0.2 of the evaluating point =0.2 plot software of the 
evaluating point =1.7 e-mail client of the semantic representation "a movie (that)", "an appreciation 
meeting (that)", "information (**)", and "creation (I want to carry out)" when the dictionary of drawing 
2 is used to a word processor — as — the evaluating point for every software object is acquired, and a 
"word processor" with the highest evaluating point is chosen as optimal software object. After passing 
through the step which acquires acknowledgement from a user, you may make it choose a software 
object here, although a software object with the highest evaluating point may be chosen automatically. 
[0012] Next, the environment for operating the software object chosen as mentioned above is set up 
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(step 53). Semantic representa^^is translated into a functional descripti(^Jcpression using the 
dictionary (it is hereafter called a functional translation section dictionary) for specifically translating 
into the language which had the function of a software object normalized. An example of a functional 
translation section dictionary is shown in drawing 3 . The functional translation section dictionary 50 of 
drawing.! is the conversion table which made the input word and the output word (translation) which 
can be replaced one conversion pair. "Creation" which is for example, an input word is convertible for 
the output word of "making" with this conversion table. In the example of drawin g 3 , further, when 
there is a candidate of two or more output words to one input word, the evaluating point which shows 
the class of input word and the validity of conversion to an output word as additional information for 
choosing a suitable output word is given to each conversion pair. This evaluating point changes 
dynamically in process of an activity. 

[0013] The conversion (translation) procedure from the input word based on the dictionary of drawing 3 
to an output word is explained still more concretely. For example, if the word "creation" is seen, this 
language will be first translated for "making." Although it can translate "it makes" into either "a 
document is made", "a graphic form being made" and "e-mail being made" further, since the word 
processor is chosen as a software object here, the evaluating point of "making a document" is the 
highest. Therefore, "a document is made" is automatically chosen as a translation (translation) (or after 
acquiring acknowledgement from a user). "A document is made" is further translated for "newly 
drawing up the /document which starts a word processor." The entry which corresponds for "newly 
drawing up the /document which starts a word processor" does not exist in a dictionary. Therefore, it 
becomes the functional technical expression of "creation" "which newly draws up the /document which 
starts a word processor." similarly, if each word of the semantic representation "a movie (that)", "an 
appreciation meeting (that)", "information (**)", and "creation (I want to carry out)" is translated 
recursively, finally functional description expression of "starting a word processor", "guidance of a 
movie appreciation meeting", and "newly drawing up a document" up will be obtained. 
[0014] Next, the operating instructions of a software object are generated from the above-mentioned 
functional description expression, and the instruction is executed (step 54). For example, the instruction 
description train for reading and performing the program of the word processor saved in the 
predetermined location of a hard disk to the functional description expression of "starting a word 
processor" is generated, and OS is performed. Moreover, to the functional description expression of 
"newly drawing up a document", the instruction description train for calling a document new creation 
function is generated, and the program of a word processor is performed through OS. The instruction 
description train which should be passed to OS is created according to the specification of the 
application programming interface (API) of OS, and the instruction description train which should be 
passed to the program of a word processor is created according to the specification of API of a word 
processor. The script for using various functions the command line for starting a program and inside a 
working program environment as an example of an instruction description train is mentioned. 
[0015] Next, the result obtained by performing an instruction description train by OS or the software 
object is outputted with the predetermined gestalt which a user can recognize. For example, when the 
instruction which corresponds for "starting a word processor" is able to perform normally, the window 
of a word processor is displayed on a forefront side on the screen of a computer (step 55). Moreover, 
when the instruction which corresponds for "newly drawing up a document" is able to perform normally, 
the document of a blank paper is generated in the window of a word processor. In addition, when 
actuation is not able to perform normally, predetermined error processing is performed (step 56). 
[0016] 

[Effect of the Invention] As mentioned above, this invention chooses automatically the optimal software 
object for performing processing according to the demand of a user inputted with natural language, and 
offers the fundamental architecture which generates automatically the suitable instruction description 
train for operating the software object further. Cooperation with a software object and a natural language 
interface is more easily [ than before ] realizable with such this invention. That is, while defining the 
instruction description train for operating a software object, the structure which operates the software 



http : // www4 . ipdl . ncipi . go .j p/cgi-bin/fr an_web_cgi_ej j e 



12/14/2004 



Page 4 of 7 

object with natural language il^fcizable only by creating the dictionary J^vhich associate each 
instruction description train and a functional description expression. 

[0017] Moreover, to catching the character string of the inputted natural language with an instruction 
(namely, thing equivalent to the functional description expression of this invention) of a user, by this 
invention, it catches with the expression of a user's demand of the character string of the inputted natural 
language, a character string is analyzed, and the functional description expression of a software object is 
generated in-between at a Prior art using various dictionaries. In other words, a user is made to express 
what he wants to carry out by means of language by this invention to having made the user express how 
he wants to use the function of a software object conventionally by means of language. Therefore, even 
if a user does not know in advance what kind of software object can be used or with what kind of 
function each software object is equipped, he only expresses in direct language what he wants to carry 
out, and can operate a software object. 
[0018] 

[Example] The outline configuration of an example of computer system equipped with the natural 
language interface constituted according to this invention is shown in drawin g 4 . This computer system 
was constituted using the common personal computer, and is equipped with the network controller 15, 
the user interface adapter 16, the display controller 21, and display 22 for cooperating with CPU 
(arithmetic and program control) 10, ROM (read-only main storage)! 1, RAM (read-and- write possible 
main storage) 12, the external storage controller 13, and external storage (auxiliary storage unit) 14 and 
the exterior. The various input devices (a keyboard 17, the microphone 18 for voice input, a mouse 19, 
tablet 20 for a handwriting input) for inputting a language train are connected to the user interface 
adapter 16. 

[0019] The functional configuration of the system of this example is shown in drawing 5 . In drawing 5 , 
the natural language input section 30 is a means to generate the character string which receives the input 
of the language by natural language, a language train, or a sentence (these are hereafter called 
"language" collectively), and expresses the language. It can choose from actuation of the alphabetic 
character input panel on the screen using the key input using the keyboard 1 7 as the input approach of 
language, the voice input using a microphone 18, and a mouse 19, or the handwriting input using a tablet 
20. of course — approaches other than these ~ the input device and the software of a sake for it (driver) - 
- even preparing — if it carries out, it can use for the input of language. 

[0020] The natural language analysis section 34 has natural language analysis, parsing in a dictionary, 
dialogic-operation functor generation, and a category dictionary manager function, analyzes the above- 
mentioned character string and generates semantic representation. The technique generally known for 
the field of natural language processing can be used for the analysis of a character string. For example, 
although natural language analytic engines of the "bamboo whisk" of Nara Institute of Science and 
Technology and Kyoto University, such as "KNP", are already known, the natural language analysis 
section 34 can be constituted using such an existing engine. 

[0021] The configuration section 36 searches the configuration section dictionary 39 (refer to drawing 
2 ) with all the concepts that appear in semantic representation, chooses the optimal software object for 
performing processing according to a demand of a user, and sets up the operating environment of the 
software object. In the configuration section dictionary 39, the information which associates an available 
software object, and the information about the configuration approach of a software object are saved in 
the concept and system which are used for semantic representation. A setup of the dictionary used for 
subsequent processings and a setup of the environment in the device by which a software object operates 
are included in configuration. When the configuration approach is described with natural language, a 
series of actuation is recursively repeated through the natural language analysis section 34. 
[0022] The functional translation section 37 searches the functional translation section dictionary 40 
(refer to drawing 3 ) with all the concepts that appear in semantic representation, and transposes it to the 
functional description expression suitable for the function of the software object memorized by the 
dictionary. Since the natural language itself may be registered into the dictionary, this permutation 
processing turns into recursive processing through the natural language analysis section 34. The 
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functional description expressSRnally generated turns into semantic replantation which consists of 
the normalized language. In addition, when the undefined part is defined in the dictionary, the functional 
translation section 37 receives assignment of an undefined part from a user through the user dialogue 
section 3 1 . 

[0023] The instruction transfer section 38 searches the instruction transfer section dictionary 41 with all 
the concepts that appear in the functional description expression created by the functional translation 
section 37, and generates the instruction description train for performing the function of the software 
object 42 memorized by the dictionary. An instruction description train is series of commands passed 
through API, parameter, or command stream of the applicable software object 42. The instruction 
transfer section 38 performs an instruction description train, and performs the function of the applicable 
software object 42. 

[0024] The response generating section 33 answers with the gestalt in which reception and a user ask for 
the activation result of the software object 42 performed by the instruction transfer section 38. The 
gestalt of a response can consider various things, such as a display on a display 22, printing by the 
printer (not shown), information storing in a database, and control of a device. The result obtained by 
performing the function of the software object 42 is inadequate, and when it cannot answer with the 
gestalt for which the user asked, the response generating section 33 shows a user a message through the 
user dialogue section 3 1 , and if there is need, it will ask a user for directions. 

[0025] The dictionary Management Department 35 performs creation of the new information on the 
configuration section dictionary 39, the functional translation section dictionary 40, and the instruction 
transfer section dictionary 41, modification of storing information, deletion, and a perusal display. 
Furthermore, a control section 42 delivers data required for the natural language input section 30, the 
natural language analysis section 34, the configuration section 36, the functional translation section 37, 
the instruction transfer section 38, the response generating section 33, the user dialogue section 31, and 
the dictionary Management Department 35, and controls a series of actuation. 

[0026] It explains referring to drawing 1 - drawing 3 about the procedure of processing the input string 
of "wanting to create information of a movie appreciation meeting" by the system of this example. 
[0027] If the user who will create information of a movie appreciation meeting and who thought drives 
in the sentence "want to create information of a movie appreciation meeting" using a keyboard 24, the 
natural language input section 30 will receive the character string of "wanting to create information of a 
movie appreciation meeting" through a keyboard entry interface (step 50). This character string is passed 
to the natural language analysis section 34. 

[0028] the natural language analysis section 34 generates the semantic representation which analyzes the 
received character string, for example, consists of the functor and four words separated semantically "a 
movie (that)", "an appreciation meeting (that)", "information (**)", and "creation (I want to carry 
out)" (step 51). This semantic representation is passed to the configuration section 36. 
[0029] The configuration section 36 acquires the evaluating point classified by software object of the 
four above-mentioned language based on the configuration section dictionary 39 (refer to drawin g 2 ), 
and the highest software object of a synthetic evaluating point judges it to be a "word processor", and it 
performs configuration processing to a "word processor" word processor [ which is stored in the 
configuration section dictionary 39 ] Turn (step 52). A setup of the functional translation section 
dictionary 40 and the instruction transfer section dictionary 41, the check of a computer resource, and 
reservation are included in configuration processing. 

[0030] The functional translation section 37 translates semantic representation into a functional 
description expression by replacing in the function in which a software object offers the four above- 
mentioned language based on the functional translation section dictionary 40 (refer to drawing 3 ), and 
its combination (step 53). For example, although there is a candidate of two output words (translation), 
"a graphic form is made", about "it makes", since the evaluating point of "making a document" is the 
highest here, it changes for "making a document". [ "a document is made" and ] in this way, "the movie 
(that)" on which the natural language interpretation section 34 generated the functional translation 
section 37 — the functional translation section dictionary 40 of drawin g 3 searches and permutes 
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recursively the semantic represWKtion "an appreciation meeting (that)", "l^Wrmation (**)", and 
"creation (I want to carry out)", and "a word processor is started", "guidance of a movie appreciation 
meeting", and the functional description expression of "newly drawing up a document" are generated. At 
the time of recursive retrieval and a permutation, semantic representation is dynamically changed using 
the natural language analysis section 34. 

[0031] Next, the instruction transfer section 38 generates an instruction description train using the 
instruction transfer section dictionary 41 (step 54). if it takes for the example "which starts a word 
processor", first, the natural language analysis section 34 will analyze this character string, and will 
separate "a word processor (**)" and "starting (it carries out)." Next, the instruction transfer section 38 
searches the instruction transfer section dictionary 41 with these concepts, and generates an instruction 
description train. In this case, "starting (it carries out)" replaces the software which starts the specific 
application for word processors through API which an operating system offers and which can be 
performed, and the instruction transfer section 38 performs. Generation of an instruction description 
train also performs retrieval and a permutation recursively, and changes semantic representation 
dynamically using the natural language analysis section 34. 

[0032] Next, the response generation section 33 checks what the word processor started, and arranges a 
word processor to the forefront side of a display (step 55). In addition, when a word processor does not 
start according to a certain failure, the response generation section 33 converses with a user through the 
user dialogue section 31, and determines a solution (step 56). If a word processor starts, a user will 
create a document by carrying out the sequential input of the thing (demand) which he wants to do 
succeedingly by means of language. For example, the language inputted says "a title is made 
information of a movie appreciation meeting", and "I want to emphasize a title." In addition, an input of 
the word of "ending" terminates a program. 

[0033] In said example, "information of a movie appreciation meeting" was created because a user 
performs a series of natural language inputs. Next, by registering into a system the operating procedure 
which creates this "information" explains the procedure which enables it to re-create same "information" 
simply. Here, it thinks of enabling it to create as an example "information" which changed introduction 
of time, a location, a movie name, and a movie freely. 

[0034] First, the functional description expression equivalent to a series of actuation in which it 
explained previously is suitably registered into the functional translation section dictionary 40 by the 
identifier (here, it considers as "information of a movie appreciation meeting") through the dictionary 
Management Department 35. Next, time, a location, a movie name, and the part equivalent to 
introduction of a movie are redefined as an undefined part through the dictionary Management 
Department 35 among the character strings contained in said a series of functional description 
expressions registered into the functional translation section dictionary 40 (step 60). 
[0035] Next, through the dictionary Management Department 35, the character string "information of a 
movie appreciation meeting" is related with a word processor software object, and it registers with the 
configuration section dictionary 39 (step 61). 

[0036] If a user performs a natural language input, saying "information of a movie appreciation meeting 
being created" through the natural language input section 30 after adding the entry of "information of a 
movie appreciation meeting" to the functional translation section dictionary 40 and the configuration 
section dictionary 39 as mentioned above, the natural language analysis section 34 and the configuration 
section 36 will perform the same processing as the example of drawing 1 , and will generate semantic 
representation (step 62). 

[0037] Next, in case the functional translation section 37 translates said semantic representation into a 
functional description expression, it transposes to a series of functional description expressions which 
registered "information of a movie appreciation meeting" into the functional translation section 
dictionary 40 previously (step 63), and performs recursive translation processing which was previously 
explained to the functional description expression (step 64). If the undefined part (introduction of time, a 
location, a movie name, and a movie) contained in a functional description expression is found (step 65), 
the functional translation section 37 will ask a user about the definition of the part through the user 
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dialogue section 3 1 . If a user i^ms the language (character string) equival^rc to a definition, the 
functional translation section 37 will replace an undefined part in the language (step 66). Thus, along 
with guidance of the user dialogue section 31, a user is inputting introduction of time, a location, a 
movie name, and a movie, and can create information of a movie appreciation meeting easily. 
[0038] As mentioned above, although the example of this invention was explained, an example is not 
restricted above. For example, although two or more application software objects installed in the 
personal computer shall be operated through a natural language interface in the above-mentioned 
example, it is also possible to enable it to operate two or more electronic equipment (for a computer to 
also be included) corresponding to a network connected to networks, such as LAN and the Internet, 
through the natural language interface of a control equipment connected to this network. The system of 
forming one voice input mold controller to the mold electric appliances corresponding to a network 
which followed, for example, were connected to domestic LAN is also realizable. 



[Translation done.] 
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* NOTICES * 




JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DRAWINGS 
[D rawin g 1] 
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[Drawing 2] 
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[Drawing 4] 
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[Drawing 6] 
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